Differentiability of Distance Functions and a Proximinal Property Inducing Convexity

In a normed linear space X, consider a nonempty closed set K which has the property that for some r > 0 there exists a set of points xo € X\K, d(xoK) > r, which have closest points p(xo) € K and where the set of points xo — r((xo — p(xo))/\\xo — p(zo)||) is dense in X\K. If the norm has sufficiently strong differentiability properties, then the distance function d generated by K has similar differentiability properties and it follows that, in some spaces, K is convex. Given a real normed linear space X, a subset K is called a proximinal ( Chebyshev) set if for each x E X\K there exists a (unique) p(x) E K such that ||x p(x)|| = d(x,K) = d(x). It has long been known that in smooth finite-dimensional normed linear spaces every Chebyshev set is convex. In such spaces the metric projection x i—> p(x) is continuous on X\K and this fact is used in the proof. So it has been natural to assume the continuity of the metric projection when attempting to prove the convexity of Chebyshev sets in smooth infinite-dimensional spaces. The best result so far has been given by Vlasov [10, 11] who showed that in a Banach space with rotund dual, Chebyshev sets which have continuous metric projection are convex. For many years it has been a matter of speculation whether there exist nonconvex Chebyshev sets in Hubert space [8]; both Vlasov and Asplund [1] showed that in Hilbert space a Chebyshev set with continuous metric projection is convex. The continuity of the metric projection was shown to be not necessary when a smooth and rotund space isomorphic to Hilbert space was exhibited containing a Chebyshev subspace with discontinuous metric projection [4]. However, an examination of Vlasov's proof [7, p. 238] shows that he uses the continuity of the metric projection only to establish a differentiability property of the distance function generated by the set. Moreover stated in terms of a differentiability condition on the distance function, reference to a proximinal condition can be removed. In fact, Vlasov's Theorem can be stated as follows. Received by the editors May 14, 1987 and, in revised form, August 15, 1987. 1980 Mathematics Subject Classification (1985 Revision). Primary 41A65, 46B20.